Quantifying IT forecast quality

In this paper, we showed how to quantify the quality of IT forecasts based on Boehm’s cone of uncertainty and DeMarco’s Estimating Quality Factor. With these, we support decision making by providing critical information on IT forecasting quality to IT governors. We illustrated that plotting forecast to actual ratios against a predefined referential conical shape reveals potential biases, for instance political, involved with IT forecasting. The Estimating Quality Factor quantifies the deviation of forecasts from their actual value. Using simulations, we showed that the conical shape of Boehm’s cone is not caused by improved estimation, but can also be found when estimation accuracy decreases. We illustrated our approach by applying it to four real-world case studies (1741 projects, 12187 forecasts, 1059 million Euro). Finally, we surveyed benchmarks related to forecasting and proposed new benchmarks based on our extensive data. Most forecasting benchmarks in the literature turned out to have an unknown bias. As a consequence, we argued that such figures including Standish’s project success benchmarks are meaningless.